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MULTIPLE LEVEL MINIMUM LOGIC NETWORK 
Coke S. Reed 

FIELD OF INVENTION 

5 The present invention relates to interconnection structures for computing 

and communication systems. More specifically, the present invention relates to 
multiple level interconnection structures in which control and logic circuits are 
minimized. 

BACKGROUND OF THE INVENTION 

0 Many advanced computing systems, including supercomputers for example, 

utilize multiple computational units to improve performance in what is called a 
parallel system. The system of interconnections among parallel computational 
units is an important characteristic for determining performance. One technique 
for interconnectmg parallel computational units involves construction of a 

5 communication network similar to a telephone network in which groups of network 
elements are connected to switching systems. The switching systems are 
interconnected in a hierarchical manner so that any switching station manages a 
workable number of connections. 

One disadvantage of a network connection is an increase in the latency of 
0 access to another computational unit since transmission of a message traverses 
several stages of a network. Typically, periods of peak activity occur in which the 
network is saturated with numerous messages so that many messages 
simultaneously contend for the use of a switching station. Various network types 
have been devised with goals of reducing congestion, improving transmission 
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speed and achieving a reasonable cost. These goals are typically attained by 
rapidly communicating between nodes and minimizing the number of 
interconnections that a node must support. 

One conventional interconnection scheme is a ring of nodes with each node 
5 connected to two other nodes so that the line of interconnections forms a circle. 
The definition of a ring, in accordance with a standard definition of a ring network 
in the art of computing (IBM Dictionary of Computing . McDaniel G. ed., 
McGraw-Hill, Inc., 1994, p. 584) is a network configuration in which devices are 
connected by unidirectional transmission links to form a closed path. Another 

10 simple conventional scheme is a mesh in which each node is connected to its four 
nearest neighbors. The ring and mesh techniques advantageously limit the number 
of interconnections supported by a node. Unfortunately, the ring and mesh 
networks typically are plagued by lengthy delays in message communication smce 
the number of nodes traversed in sending a message from one node to another may 

15 be quite large. These lengthy delays commonly cause a computational unit to 
remain idle awaiting a message in transit to the unit. 

The earliest networks, generally beginning with telephone networks, utilize 
circuit switching in which each message is routed through die network along a 
dedicated path that is reserved for the duration of the commrmication analogous to 

20 a du-ect connection via a single cu-cuit between the communicatLng parties. Circuit 
switching disadvantageously requires a lengthy setup time. Such delays are 
intolerable during the short and quick exchanges that take place between different 
computational units. Furthermore, a dedicated pathway is very wasteful of system 
bandwidth. One technique for solving the problems arising using circuit switching 

25 is called packet switching in which messages sent from one computational unit to 
another does not travel in a contmuous stream to a dedicated circuit. Instead, each 
computational unit is connected to a node that subdivides messages into a sequence 
of data packets. A message contains an arbitrary sequence of binary digits that are 
preceded by addressing infomiation. The length of the entire message is limited 
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to a defined maximum length. A "header" containing at least the destination 
address and a sequence number is attached to each packet, and the packets are sent 
across the network. Addresses are read and packets are delivered within a fraction 
of a second. No circuit setup delay is imposed because no circuit is set up. 
5 System bandwidth is not wasted since there is no individual cormection between 
two computational units. However, a small portion of the communication capacity 
is used for routing information, headers and other control information. When 
communication advances in isolated, short bursts, packet switching more efficiently 
Utilizes network capacity. Because no transmission capacity is specifically 
10 reserved for an individual computational unit, time gaps between packets are filled 
with packets from other users. Packet switching implements a type of distributed 
multiplexing system by enabling all users to share lines on the. network 
continuously. 

Advances in technology result in improvement in computer system 
15 performance. However, the manner in which these technological advances are 
implemented will greatly determine the extent of improvement in performance. 
For example, performance improvements arising from completely optical 
computing strongly depend on an interconnection scheme that best exploits the 
advantages of optical technology. 

20 SUMMARY OF THE INVENTION 

In accordance with the present invention, a multiple level minimum logic 
network interconnect structure has a very high bandwidth and low latency. 
Control of intercoimect structure switching is distributed throughout multiple nodes 
in the structure so that a supervisory controller providing a global control function 
25 is not necessary. A global control function is eliminated and complex logic 
structures are avoided by a novel data flow technique that is based on timing and 
positioning of messages communicating through the interconnect structure. 
Furthermore, the interconnect structure implements a "deflection" or "hot potato" 
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design in which processing and storage overhead at each node is minimized by 
routing a message packet through an additional output port rather than holding the 
packet until a desired output port is available. Accordingly, the usage of buffers 
at the nodes is eluninated. Elimination of a global controller and buffermg at the 
5 nodes greatly reduces the amount of control and logic structures in the mterconnect 
structure, simplifying overall control components and network interconnect 
components, improvmg speed performance of message communication and 
potentially reducing interconnection costs substantially. Implementation of the 
interconnect structure is highly flexible so that fiilly electronic, folly optical and 
10 mixed electronic-optical embodiments are achieved. An implementation using all 
optical switches is facilitated by nodes exploiting uniquely sraiple logic and 
elimination of buffering at the nodes. 

The multiple level minimum logic network interconnect architecture is used 
for various purposes. For example, in some embodiments the architecture is used 

15 as an interconnect structure for a massively parallel computer such as a 
supercomputer. In other exemplary embodiments, the architecture forms an 
interconnect structure Imking a group of workstations, computers, terminals, ATM 
machines, elements of a national flight control system and the like. Another usage 
is an interconnect structure m various telecommunications applications or an 

20 interconnect structure for numerous schedulers operating in a business main frame. 

In accordance with one aspect of the present invention, an interconnect 
apparatus includes a plurality of nodes and a plurality of interconnect lines 
selectively connectmg the nodes in a multiple level strucmre in which the levels 
mclude a richly interconnected collection of rings. The multiple level structure 
25 includes a plurality of J+1 levels in a hierarchy of levels and a plurality of 2^K 
nodes at each level. If integer K is an odd number, the nodes on a level M are 
situated on 2^''** rings with each ring including 2^K nodes. Message data leaves 
the interconnect structure from nodes on a level zero. Each node has multiple 
communication termmals. Some are message data input and output terminals. 
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Others are control input and output terminals. For example, a node A on level 0, 
the innermost level, receives message data from a node B on level 0 and also 
receives message data from a node C on level 1. Node A sends message data to 
a node D on level 0 and also sends message data to a device E that is typically 
5 outside the interconnect structure. One example of a device E is an input buffer 
of a computational unit. Node A receives a control input signal from a device F 
which is commonly outside the intercormect structure. An example of a device F 
is an^m^i buffer of a computational unit. Node A sends a control signal to a 
node G on level 1. ^ \ ^^^J 

10 All message data enters the interconnect stmcmre on an outermost level J. 

For example, a node A on level J, the outermost level, receives message data from 
a node B on level J and also receives message data from a device C that is outside 
the interconnect structore. One example of device C is an output buffer of a 
computational unit. Node A sends message data to a node D on level J and also 

15 sends message data to a node E on level J-1. Node A receives a control input 
signal from a node F on level J-1. Node A sends a control signal to a device G 
that is typically outside the interconnect structore. An example of a device G is 
an output buffer of a computational unit. 



Nodes between the iimermost level 0 and the outermost level J 
20 communicate message data and control signals among other nodes. For example, 
a node A on a level T that is neither level 0 or level J receives message data from 
a node B on level T and also receives message data from a node C on level T + 1. 
Node A sends message data to a node D on level T and also sends message data 
to a node E on level T-1. Node A receives a control input signal from a node F 
25 on level T-1. Node A sends a control signal to a node G on level T -t-1. 

Level M has 2^'^ rings, each containing 2^K nodes for a total of 2^K nodes 
on level M. Specifically: 
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Level 0 has 2^ rings, each containing 2°K = K nodes for a total of 
2^K nodes on level 0. 

Level 1 has 2^'^ rings, each containing 2^K = 2K nodes for a total 
of 2^K nodes on level 1 . 
5 Level 2 has 2^'^ rings, each containing 2^K = 4K nodes for a total 

of 2^K nodes on level M. 



10 Level J-2 has 2^"^^"^' = 4 rings, each containing 2^^'^^K nodes for a 

total of 2^K nodes on level J-2. 

Level J-1 has 2^"<^-^> = 2 rings, each containing 2^^-^)K nodes for a 
total of 2^K nodes on level J-L 

Level J has 2^"^ = 1 ring containing 2^^''^'^K nodes for a total of 2^K 
15 nodes on level J. 

For a ring Rj on a level T which is not the outermost level J, then one ring 
Rt+1 on level T+1 exists such that each node A on ring receives data from a 
node B on ring and a node C on ring R^+i- For a ring R^ on a level T which 
is not the innermost level 0, then there exist exactly two rings Rlj.i and Rlj.i on 
20 level T-1 such that a node A on ring Rj sends message data to a node D on ring 
Rj and a node E on either ring Rlx-i or ring R2r.i. A message on any level M 
of the interconnect structure can travel to two of the rings on level M-1 and is 
eventually able to travel to 2^ of the rings on level 0. 

In the following discussion a "predecessor" of a node sends message data 
25 to that node. An "immediate predecessor" sends message data to a node on the 
same rmg. A "successor" of a node receives message data from that node. An 
"immediate successor" receives message data to a node on the same ring. 
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For a node Arx on ring on level T, there are nodes Brt and D^^ on 
ring R-j. of level T such that node Br-j- is an immediate predecessor of node A^^- 
and node Drj is an immediate successor of node Art- Node Ar^ receives message 
data from node Br-j. and sends message data to node Dr^. Node Ar-j. receives 
5 message data from a device C that is not on the ring and sends data to a device 
E that is not on ring R^. If the level is not the innermost level 0, then device E 
is a node on level T-1 and there is an unmediate predecessor node F on the same 
ring as device E. Node Arj receives control information from device F. If node 
Art is on node T equal to zero, then device E is outside the interconnect structure 
10 and device E sends control information to node Art- For example, if device E is 
an input buffer of a computational unit, then the control information from device 
E to node Arj indicates to node Art whether device E is ready to receive message 
data from node Arj. Node Dr-t receives message data from a device G that is not 
on ring Rj. Node Ar-j- sends a control signal to device G. 

15 Control information is conveyed to resolve data transmission conflicts in 

the interconnect structure. Each node is a successor to a node on the adjacent 
outer level and an immediate successor to a node on the same level. Message data 
from the iimnediate successor has priority. Control information is send from 
nodes on a level to nodes on the adjacent outer level to warn of impending 

20 conflicts. 

When the levels are evenly spaced and the nodes on each ring and each 
level are evenly spaced, the interconnect structure forms a three-dimensional 
cylindrical structure. The interconnect structure is fully defined by designating the 
interconnections for each node A of each level T to devices or nodes B, C, D, E, 
25 F and G. Each node or device has a location designated in three-dimensional 
cylindrical coordinates (r, 6, z) where radius r is an integer which specifies the 
cylinder number from 0 to J, angle 6 is an integer multiple of 27r/K, which 
specifies the spacing of nodes around the circular cross-section of a cylinder from 
0 to K-1, and height z'is a binary integer which specifies distance along the z-axis 
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from 0 to 2^-1. Height z is expressed as a binary number because the 
interconnection between nodes in the z-dimension is most easily described as a 
binary digit manipulation. On the innermost level 0, one ring is spanned in one 
pass through the angles 6 from 0 to K-1 and each height z designates a ring. On 
5 level 1, one ring is spanned in two passes through the angles 6 and two heights z 
are used to designate one ring. The ring structure proceeds in this manner through 
die outermost ring J in which one ring is spanned in all 2^ heights along the z-axis. 

Node A on a ring R receives message data from a node B, which is an 
immediate predecessor of node A on ring R. For a node A located at a node 

10 position N(r,^,z), node B is positioned at N(r,(^-l)mod K,U^(z)) on level r. (6- 
l)mod K is equal(K}yvhen 6 is equal to 0 and equal to ^-1 otherwise. The 
conversion of z to tLjiz) on a level r is described for z = [zj.^, Zj.2, . . , z^, z^.j, 
. . , Z2, Zi, Zfl] by reversing the order of low-order z bits from Zj..i to Zq] into the 
form z = [Zj.i, Zj.2, - - , z^, Zq, z^, Z2, . z^.J, subtracting one (modulus 20, and 

15 reversing back the modified low-order z bits. JU»c^ 

Node A also receives message data from a device C which is not on level 
r. If node A is positioned on the outermost level r=J, then device C is outside of 
the interconnect structure. If node A is not positioned on the outermost level, then 
device C is a node located at position N(r+l,(^-l)niod K,z) on level r+l. 



20 Node A sends message data to a node D, which is an immediate successor 

to node A on ring R. Node D is located at node position N(r,(0 + l)mod K,h^(z)) 
on level r. (0+l)mod K is equal 0 when 6 is equal to K-1 and equal to ^+1 
otherwise. The conversion of z to \(z) on a level r is described for z = [zj.j, Zj.2, 
. . , Zr, Zj.i, . . , Z2, Zi, Zq] by reversing the order of low-order z bits from 7^_i to 

25 Zq] into the form z = [Zj.^, Zj.j, . . , z^, Zq, Zj, Z2, . z^.i], adding one (modulus 
2") and reversing back the low-order z bits. 
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Node A also sends message data to a device E that is not on the same level 
r as node A. If node A is on the innermost level r=0, node A(t,6,z) is 
interconnected with a device (e.g. a computational unit) outside of the interconnect 
structure. Otherwise, node A is interconnected to send message data to device E, 
5 which is a node located at node position N(r-l,(e+l)mod K,z) on level r-l. 

Node A receives control information from a device F. If node A is on the 
innermost level r=0, the device F is the same as device E. If node A is not on 
the innermost level, device F is a node which is distinct from the device E. Node 
10 F is located at node position N(r-l,^,Hj.i(z)) on level r-l. 

Node A sends control information to a device G. If node A is on the 
outermost level r=J, then device G is positioned outside of the interconnect 
structure. Device G is a device, for example a computational unit, that sends 
message data to node D. If node A is not positioned on level r=J, then device G 
15 is a node which is located at node position N(r+l,^,hj+i(z)) on level r+l and 
device G sends message data to node D. 

In accordance with a second aspect of the present invention, a method is 
shown of transmitting a message from a node N to a target destination in a first, 
a second and a third dimension of three dimensions in an interconnect structure 

20 arranged as a plurality of nodes in a topology of the three dunensions. The 
method includes the steps of determining whether a node en route to the target 
destination in the first and second dunensions and advancing one level toward the 
destination level of the third dimension is blocked by another message, advancing 
the message one level toward the destination level of the tMrd dimension when the 

25 en route node is not blocked and moving the message in the first and second 
dimensions along a constant level in the third dimension otherwise. This method 
further includes the step of specifying the tWrd dimension to describe a plurality 
of levels and specifying the first and second dimensions to described a plurality of 
nodes on each level. A control signal is sent from the node en route to the node 
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N on a level q in the third dimension, the control signal specifying whether the 
node en route is blocked. Transmission of a message is timed using a global clock 
specifying timing intervals to keep integral time modulus the number of nodes at 
a particular cylindrical height, the global clock time interval being equal to the 
5 second time interval and the first time interval being smaller dian the global time 
mterval. A first time interval a is set for moving the message in only the first and 
second dimensions. A second time interval or - /S is set for advancing the message 
one level toward the destination level. A third time interval is set for sending the 
control signal from the node en route to the node N, the third time interval being 
10 equal to /5. 

In accordance with a diird aspect of the present invention, a method is 
shown of transmitting a message from an input device to an output device through 
an interconnect structure. The message travels through the interconnect structure 
connecting a plurality of nodes in a three dunensional structure. The message has 

15 a target destination corresponding to a target ring on level 0 of the interconnect 
structure. A message M at a node N on level T en route to a target ring on level 
0 advances to a node N' on level T-1 so long as the target ring is accessible from 
node N' and no other higher priority message is progressmg to node N' to block 
the progress of message M. Whether the target ring is accessible from node N' 

20 is typically efficiently determined by testmg a smgle bit of a binary code 
designating the target ring. Whether a higher priority message is blocking the 
progress of message M is efficiently determined using timed control signals. If 
a message is blocked at a tune t, the message is in position to progress to the next 
level at time t+2. If a message is blocked by a message M' on levej T-1, then a 

25 limited time duration will transpire before the message M ' is able to block message 
M agam. 

A global clock controls traffic flow in the interconnect strucmre. Data flow 
follows rules that allow much of the control information to be "hidden" in system 
timing so that, rather than encodmg all control information m a message packet 
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header, timing considerations convey some information. For example, the target 
ring is encoded in the message packet header but, in some embodiments of the 
interconnect structure, designation of the target computational unit is determined 
by the timing of arrival of a message with respect to time on the global clock. 

5 The disclosed multiple level interconnect structure has many advantages. 

One advantage is that the structure is simple, highly ordered and achieves fast and 
efficient communication for systems having a wide range of sizes, from small 
systems to enormous systems. 

In addition, the interconnect structore is highly advantageous for many 
10 reasons. The interconnect structure resolves contention among messages directed 
toward the same node and ensures that a message that is blocked makes a complete 
tour of the messages at a given angle on a level before the blocking message is in 
position to block again. In this manner, a message inherently moves to cover all 
possible paths to the next level. A blocking message typically proceeds to 
15 subsequent levels so that overlying messages are not blocked for long. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The features of the invention believed to be novel are specifically set forth 
in the appended claims. However, the invention itself, both as to its structore and 
method of operation, may best be understood by referring to the following 
20 description and accompanying drawings. 

Figures lA, IB, IC and ID are abstract three-dimensional pictorial 
illustrations of the structure of an embodiment of a multiple level minimum logic 
interconnect apparatus. 

Figure 2 is a schematic diagram of a node, node terminals and 
25 interconnection lines connected to the terminals. 
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Figures 3A, 3B and 3C are schematic block diagrams that illustrate 
interconnections of nodes on various levels of the interconnect structure. 

Figure 4 is an abstract schematic pictorial diagram showing the topology 
of levels of an interconnect structure. 

Figure 5 is an abstract schematic pictorial diagram showing the topology 
of nodes of an interconnect structure. 

Figure 6 is an abstract schematic pictorial diagram which illustrates the 
manner in which nodes of the rings on a particular cylindrical level are 
interconnected. 

Figure 7 illustrates interconnections of a node on level zero. 

Figure 8 depicts interconnections of a node on level one. 

Figure 9 depicts interconnections of a node on level two. 

Figure 10 depicts interconnections of a node on level three. 

Figure 11 is an abstract schematic pictorial diagram which illustrates 
interconnections between devices and nodes of a rmg on the low level cylinder. 

Figure 12 is an abstract schematic pictorial diagram which illustrates 
interconnections among nodes of two adjacent cylindrical levels. 

Figure 13 is an abstract schematic pictorial diagram showing 
interconnections of nodes on cylindrical level one. 
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Figure 14 is an abstract schematic pictorial diagram showing 
interconnections of nodes on cylindrical level two. 

Figure 15 is an abstract schematic pictorial diagram showing 
interconnections of nodes on cylindrical level three. 

5 Figure 16 is an abstract schematic pictorial diagram illustrating the 

interaction of messages on adjacent levels of an embodiment of the interconnection 
structure. 

Figure 17 is a timing diagram which illustrates timing of message 
communication in the described interconnect structure. 

10 Figure 18 is a pictorial representation illustratmg the format of a message 

packet including a header and payload. 

Figure 19 is a pictorial diagram which illustrates the operation of a lithium 
niobate node, a first exemplary node strucmre. 

Figure 20 is a pictorial diagram which illustrates the operation of a 
15 nonlinear optical loop mirror (NOLM), a second exemplary node strucmre. 

Figure 21 is a pictorial diagram which illustrates the operation of a 
terahertz optical asymmetrical demultiplexer (TOAD) switch, a third exemplary 
node structure. 

Figure 22 is a pictorial diagram showing the operation of a regenerator 
20 utilizing a lithium niobate gate. 
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Figure 23 is an abstract schematic pictorial diagram illustrating an 
alternative embodiment of an intercomiect structure in which devices issue message 
packets to multiple nodes. 

Figure 24 is an abstract schematic pictorial diagram illustratuig an 
5 alternative embodiment of an interconnect strucmre in which devices receive 
message packets from multiple nodes. 

Figure 25 is an abstract schematic pictorial diagram illustrating an 
alternative embodiment of an intercomiect strucmre in which devices issue message 
packets to nodes at various interconnect levels. 

10 DETAILED DESCRIPTION 

Referring to Figures lA, IB, IC and ID, an embodiment of a multiple 
level minimum logic interconnect apparatus 100 includes multiple nodes 102 which 
are connected in a multiple level interconnect structure by interconnect lines 104. 
The multiple level interconnect structure is shown illustratively as a three- 
15 dimensional strucmre to facilitate understanding. 

The nodes 102 m the multiple level interconnect structure are arranged to 
include multiple levels 110, each level 110 having a hierarchical significance so 
that, after a message is initiated in the structure, the messages generally move 
from an initial level 112 to a final level 114 in the direction of levels of a previous 

20 hierarchical significance 116 to levels of a subsequent hierarchical- significance 
118. Illustratively, each level 110 includes multiple strucUires which are called 
rings 120. Each rmg 120 includes multiple nodes 102. The term "rings" is used 
merely to facilitate understanding of the strucmre of a network m the abstract m 
which visualization of the structure as a collection of concentric cylindrical levels 

25 110 is useful. 
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The different Figures lA, IB, IC and ID are included to more easily 
visualize and understand the interconnections between nodes. Figure lA illustrates 
message data transmission interconnections between nodes 102 on the various 
cylindrical levels 110. Figure IB adds a depiction of message data transmission 
5 interconnections between nodes 102 and devices 130 to the interconnections 
illustrated in Figure lA. Figure IC further shows message data interconnections 
between nodes 102 on different levels. Figure ID cumulatively shows the 
interconnections shown in Figures lA, IB and IC in addition to control 
interconnections between the nodes 102. 

10 The actual physical geometry of an interconnect structure is not to be 

limited to a cylindrical structure. What is important is that multiple nodes are 
arranged in a first class of groups and the first class of groups are arranged into 
a second class of groups. Reference to the first class of groups as rings and the 
second class of groups as levels is meant to be instructive but not limiting. 

15 The illustrative interconnect apparatus 100 has a structure which includes 

a plurality of J+1 levels 110. Each level 110 includes a plurality of 2^K nodes 
102. Each level M contains 2^'^ rings 120, each containing 2^K nodes 102. The 
total number of nodes 102 in the enth-e structure is (J+1)2^K. The interconnect 
apparatus 100 also includes a plurality 2^K devices 130. In the illustrative 

20 embodiment, each device of the 2^K devices 130 is connected to a data output port 
of each of the K nodes 102 in each ring of the 2^ rings of the final level 114. 
Typically, in an interconnect structure of a computer a device 130 is a 
computational unit such as a processor-memory unit or a cluster of processor- 
memory units and input and output buffers. 

25 Referring to Figure 2, an interconnect structure 200 of a node 102 has 

three input terminals and three output terminals. The input terminals include a 
first data input terminal 210, a second data input terminal 212 and a control ir^ut 
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terminal 214. The output terminals include a first data output terminal 220, a 
second data output terminal 222 and a control output terminal 224. The data input 
and output terminals of a node communicate message data with other nodes. The 
control terminals communicate control bits with other nodes for controlling 
5 transmission of message data. The number of control bits for controlling message 
transmission is efficiently reduced since much of the logic throughout the 
interconnect structure 200 is determined by timing of the receipt of control bits and 
message data in a manner to be detailed hereinafter. Only one control bit enters 
a node and only one control bit leaves at a given time step. Messages are 
10 communicated by generating a clock signal for timing time units. Message 
transmission is controlled so that, during one time unit, any node 102 receives 
message data from only one input terminal of the data input terminals 212 and 
214. Since, a node 202 does not have a buffer, only one of the node's output 
ports is active in one time unit. 

15 Referring to Figures 3 through 16, the topology of an interconnect 

structure 300 is illustrated. To facilitate understanding, the structm-e 300 is 
illustrated as a collection of concentric cylinders in three dunensions r, 6 and z. 
Each node or device has a location designated (r, 6, z) which relates to a position 
(r, lird/K, z) in three-dimensional cylindrical coordinates where radius r is an 

20 integer which specifies the cylinder number from 0 to J, angle 0 is an integer 
which specifies the spacing of nodes around the circular cross-section of a cylinder 
from 0 to K-1, and height z is a binary integer which specifies distance along the 
z-axis from 0 to 2^-1. Height z is expressed as a binary number because the 
interconnection between nodes in the z-dimension is most easily described as a 

25 manipulation of bmary digits. Accordingly, an interconnect strucmre 300 is 
defined with respect to two design parameters J and K. 

Figures 3A, 3B and 3C are schematic block diagrams that show 
interconnections of nodes on various levels of the interconnect structure. Figure 
3A shows a node Arj 320 on a ring R of outermost level J and the 
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interconnections of node Arj 320 to node B^j 322, device C 324, node Dpj 326, 
node Er(j.i) 328, node Fr(M) 330 and device G 332. Figure 3B shows a node A^j 
340 on a ring R of a level T and the interconnections of node A^^ 340 to node B^-j- 
342, node Cj^^^+i) 344, node Dj^^- 346, node Er^^j.^) 348, node 350 and node 
5 Gr(t-^.i) 352. Figure 3C shows a node Arq 360 on a ring R of innermost level 0 
and the interconnections of node A^q 360 to node Bj^q 362, node C^i 364, node 
Dro 366, device E 368 and node Gri 372. 



In Figures 3A, 3B and 3C interconnections are shown with solid lines with 
arrows indicating the direction of message data flow and dashed lines with arrows 
indicating the direction of control message flow. In summary, for nodes A, B and 
D and nodes or devices C, E, F, G: 

(1) A is on level T. 

(2) B and C send data to A. 

(3) D and E receive data from A. 

(4) F sends a control signal to A. 

(5) G receives a control signal from A. 

(6) B and D are on level T. 

(7) B is the immediate predecessor of A. 

(8) D is the immediate successor to A. 

(9) C, E, F and G are not on level T. 

The positions in three-dimensional cylindrical notation of the various nodes 
and devices is as follows: 

(10) A is positioned at node N(r, 6, z). 

(11) B is positioned at node N(r, 6-1, Hx(z)). 

(12) C is either positioned at node N(r+1, 6-1, z) or is outside the 
interconnect structure. 

(13) D is positioned at node N(r, 6+1, h^(z)). 

(14) E is either positioned at node N(r-1, ^-1-1, z) or 

is outside the interconnect structure and the same as device F, 
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(15) F is either positioned at node N(r-1, 0, Hx.i(z)) or is outside the 
interconnect structure and the same as device E. 

(16) G is either positioned at node N(r+1, 6, hj(z)) or is outside the 
interconnect structure. 



5 In this notation, (5-l)mod K is equal K when B is equal to 0 and equal to 

Q-\ otherwise. The conversion of z to B.^{z) on a level r is described for z = [Zj.j, 
Zj.2, • . , z^, Zj.i, . . , Z2, Zj, Zq] by reversing the order of low-order z bits from 
Zf.i to Zo] into the form z = [zj.^, Zj.2, . . , Zr, Zq, Zj, Z2, . z^.J, subtracting 
(modulus 20 and reversing back the low-order z bits. Similarly, (^+l)mod K is 
10 equal 0 when 6 is equal to K-1 and equal to 6+1 otherwise. The conversion of 
z to hi.(z) on a level r is described for z = [zj.j, Zj.2, . . , z^, z^.i, . . , Z2, z^, Zq] 
by reversing the order of low-order z bits from z^.i to Zq into the form z = [zj.i, 
Zj.2, . . , Zj, Zq, Zi, Z2, . ., Zj._i], adding (modulus 2^ and reversing back the low- 
order z bits. 



15 Referring to Figure 4, concentric cylindrical levels zero 310, one 312, two 

314 and three 316 are shown for a J=3 interconnect structure 300 where level 0 
refers to the iimermost cylindrical level, progressing outward and numerically to 
the outermost cylmdrical level 3. A node 102 on a level T is called a level T 
node. 



20 An intercoimect structure has J+1 levels and 2^K nodes on each level. 

Referring to Figure 5, the design parameter K is set equal to 5 so that the 
interconnect structure 300 has four levels (J +1=3 + 1= 4) with 40 (2^K = (2^)5 
= 40) nodes on each level. 



25 



Referrmg to Figure 6, the interconnect structure is fully defined by 
designating the interconnections for each node A 530 of each level T to devices 
or nodes B 532, C 534, D 536, E 538, F 540 and G 542. 
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Node A(r,^,z) 530 is interconnected with an immediate predecessor node 
B(r,(^-l)mod K,Hr(z)) 532 on level r. If node A(r,e,z) 530 is on the outermost 
level r= J, node A(v,6,z) 530 is interconnected with a device (e.g. a computational 
unit of a computer) outside of the interconnect structure. Otherwise, node A(r,0,z) 
5 530 is interconnected with a predecessor node C(r+ 1,(^-1 )mod K,z) 534 on level 
r-f-L 

Node A(x,0,2) 530 is interconnected with an immediate successor node 
D(r,(^+l)mod K,hj.(z)) 536 on level r. If node A(r,0,z) 530 is on the innermost 
level r=0, node A(r,6,z) 530 is interconnected with a device (e.g. a computational 
10 unit) outside of the interconnect structure. Otherwise, node A{x,6,z) 530 is 
mtercomiected with a successor node E(r-l,(0 + l)mod K,z) 538 on level r-1 to 
send message data. 

If node A(r,6,z) 530 is on the innermost level r=0, node A(r,d,z) 530 is 
interconnected with a device (e.g. a computational unit) outside of the interconnect 
15 structure. Otherwise, node A(x,6,z) 530 is interconnected with a node F(r-l,0,Hr. 
i(z)) 540 on level r-1 which supplies a control input signal to node A(r,0,z) 530. 

If node A(r,^,z) 530 is. on the outermost level r=J, node A(r,0,z) 530 is 
interconnected with a device (e.g. a computational unit) outside of the interconnect 
structure. Otherwise, node A{x,6,z) 530 is interconnected with a node 
20 G(r+l,^,hj.+i(z)) 542 on level r+1 which receives a control output signal from 
A(r,0,z) 530. 

Specifically, the interconnections of a node A for the example of an 
interconnect structore with interconnect design parameters J =3 and K=5 are 
defined for all nodes on a ring. Every ring is unidirectional and forms a closed 
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curve so that the entire structure is defined by designating for each node A, a node 
D that receives data from node A. 

Referring to Figure 7 in conjunction with Figure 6, interconnections of a 
node A on level zero are shown. Node A(0,6,z) 530 is interconnected to receive 
5 message data from immediate predecessor node B(0,(6-l)mod 5,z) 532 on level 0 
and to send message data to immediate successor node D(0,(^ + l)mod 5,z) 536 on 
level 0. Although the interconnection term in the second dimension for nodes B 
and D is previously defined as Hj.(z) and \\(z), respectively, on level zero, Hr{z) 
and hj(z) are equal to z. Node A(0,^,z) 530 is also interconnected to receive 
10 message data from predecessor node C(l,(0-l)mod 5,z) 534 on level 1 and to send 
message data to a device E($,7) 538. Node A(0,6,z) 530 is uiterconnected to 
receive a control mput signal from a device F((^-l)mod 5,z) 540 and to send a 
control output signal to node G(l,^,hj(z)) 542 on level 1. 

Referring to Figure 8 in conjunction with Figure 6, interconnections of a 
15 node A on level one are shown. Node A(l ,6,z) 530 is interconnected to receive 
message data from immediate predecessor node B(l,(5-l)mod 5,Hji(z)) 532 on 
level 1 and to send message data to immediate successor node D(l,(^+l)mod 
5,hi(z)) 536 on level 1. Height z is expressed as a binary number (base 2) having 
the form [Z2,z^,z^]. For level one, when z is [zj.Zj.O] then hi(z) and Hi(z) are 
20 both [z2,zi,l]. When z is [z2,Zi,13 then hi(z) and Hi(z) are bodi [Z2,Zi,0]. Node 
A(l,6,z) 530 is also interconnected to receive message data from predecessor node 
C(2,(^-l)mod 5,z) 534 on level 2 and to send message data to successor node 
E(O,(0+l)mod 5,z) 538 on level 0. Node A(l,e,z) 530 is interconnected to 
receive a control input signal from a node F(O,0,Hi(z)) 540 on level zero and to 
25 send a control output signal to node G(2,e,h2(z)) 542 on level 2. 

Referring to Figure 9 in conjunction with Figure 6, interconnections of a 
node A on level two are shown. Node A(2,6,z) 530 is interconnected to receive 
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message data from immediate predecessor node B(2,(^-l)mod S.HxCz)) 532 on 
level 2 and to send message data to immediate successor node D(2,(^+l)mod 
5,h2(z)) 536 on level 2. Height z is expressed as a binary number (base 2) having 
the form [z2,Zi,Zq]. For level two, when z is [Z2,0,0] then h2(z) is [Z2,l,0] and 
5 H2(z) is [Z2,l,l]. When z is [Z2,0,l] then h2(z) is [zj.l.l] and HjCz) is [Z2,l,0]. 
When z is [Z2,l,0] then h2(z) is [Z2,0,I] and H2(z) is [Zj.O.O]. When z is [Z2,l,l] 
then h2(z) is [Z2,0,0] and H2(z) is [zj.O,!]. Node A(2,d,z) 530 is also 
interconnected to receive message data from predecessor node C(3,(^-l)mod 5,z) 
534 on level 3 and to send message data to successor node E(1,(0+ l)mod 5,z) 538 
10 on level 1. Node A(2,d,z) 530 is intercormected to receive a control input signal 
from a node F(l,^,H2(z)) 540 on level 1 and to send a control output signal to 
node G(3,^,h3(z)) 542 on level 3. 

Referring to Figure 10 in conjunction with Figure 6, intercoimections of 
a node A on level three are shown. Node A(3,^,z) 530 is interconnected to 

15 receive message data from immediate predecessor node B(3,(0-l)mod 5,H3(z)) 532 
on level 3 and to send message data to immediate successor node D(3,(^+l)mod 
5,h3(z)) 536 on level 3. For level three, when z is [0,0,0] then hjCz) is [1,0,0] and 
H3(z) is [l.I.l]. When z is [0,0,1] then h^iz) is [1,0,1] and H3(z) is [1,1,0]. 
When z is [0,1.0] then h3(z) is [1,1,0] and HjCz) is [1,0,0]. When z is [0,1,1] 

20 then h^iz) is [1.1,1] and UjCz) is [1,0,1]. When z is [1,0,0] then h3(z) is [0,1,0] 
and HgCz) is [0.0.0]. When z is [1,0,1] then h^(z) is [0,1,1] and H3(z) is [0,0,1]. 
When z is [1,1,0] then h3(z) is [0,0,1] and H^iz) is [0,1,0]. When z is [1,1,1] 
then h3(z) is [0,0,0] and H3(z) is [0,1,1]. Node A(3,^,z) 530 is also 
interconnected to receive message data from predecessor node C(4,(^-l)mod 5,z) 

25 534 on level 4 and to send message data to successor node E(2, (6 + l)mod 5,z) 538 
on level 2. Node A(3,^,z) 530 is interconnected to receive a control input signal 
from a node F(2,^,H3(z)) 540 on level 2 and to send a control output signal to 
node G(4,0,h4(z)) 542 on level 4. 



M-3240 PCX 

-22- 

Figure 11 illustrates interconnections between devices 130 and nodes 102 
of a ring 120 on the cylindrical level zero 110. In accordance with the description 
of the interconnect structure 200 of a node 102 discussed with respect to Figure 
2, a node 102 has three input terminals and three output terminals, including two 
5 data input terminals and one control input terminal and two data output terminals 
and one control output terminal. In a simple embodiment, a device 130 has one 
data input terminal 402, one control bit input terminal 404, one data output 
terminal 406 and one control bit output terminal 408. 

Referring to Figure 11, nodes 102 at the lowest cylindrical level 110, 
10 specifically nodes N(0,^,z), are connected to devices CU(^,z). In particular, the 
data input terminal 402 of devices C\J(d,z) are connected to the second data output 
terminal 222 of nodes N(0,^,z). The control bit output terminal 408 of devices 
CU(0,z) are connected to the control input terminal 214 of nodes N(0,^,z). 

The devices CU{d,z) are also connected to nodes N(J,^,z) at the outermost 
15 cylinder level. In particular, the data output terminal 406 of devices CU(0,z) are 
connected to the second data input terminal 212 of nodes N(l,d,z). The control 
bit ii^ut terminal 404 of devices CU(0,z) are connected to the control output 
terminal 224 of nodes N(0,d,z). Messages are communicated from devices 
CXJ(d,z) to nodes N(J,0,z) at the outermost cylindrical level J. Then messages 
20 move sequentially mward from the outermost cylindrical level J to level J-1, level 
J-2 and so forth unit the messages reach level 0 and then enter a device. Messages 
on the outermost cylinder J can reach any of the 2^ rings at level zero. Generally, 
messages on any cylindrical level T can reach a node on 2'^ rings on level zero. 

Figure 12 illustrates interconnections among nodes 102 of two adjacent 
25 cylindrical levels 110. Referring to Figure 12 in conjunction with Figure 2, nodes 
102 at the T cylindrical level 110, specifically nodes N(T,0,z) 450, have terminals 
connected to nodes on the T level, the T+1 level and the T-1 level. These 
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connections are such that the nodes N(T,^,z) 450 have one data input terminal 
connected to a node on the same level T and one data input terminal connected to 
another source, usually a node on the next outer level T-t-1 but for nodes on the 
outermost level J, a device is a source. In particular, nodes N(T,0,z) 450 have a 
5 first data input terminal 210 which is connected to a first data output terminal 220 
of nodes N(T+l,e-l,z) 452. Also, nodes N(T,e,z) 450 have a first data output 
terminal 220 which is connected to a first data input terminal 210 of nodes N(T- 
i,d+l,z) 454. 

The nodes N(T,6,z) 450 also have a second data input terminal 212 and a 

10 second data output terminal 222 which are cormected to nodes 102 on the same 
level T. The second data mput terminal 212 of nodes N(T,^,z) 450 are connected 
to the second data output terminal 222 of nodes N(T,^-l,hx(z)) 456. The second 
data output terminal 222 of nodes N(T,0,z) 450 are connected to the second data 
input termmal 212 of nodes N(T,6 + 1 ,Hi-(z)) 458 . The cylinder height designation 

15 Ht(z) is determined using an inverse operation of the technique for determining 
height designation hp(z). The interconnection of nodes from cylindrical height to 
height (height z to height Ht(z) and height hx(z) to height z) on the same level T 
is precisely defined according to a height transformation technique and depends on 
the particular level T within which messages are commimicated. Specifically in 

20 accordance with the height transformation technique, the height position z is put 
into bmary form where z = Zj.fi^'^ + Zj.jl^"^ + . . . + + •z^.{2^''^ + . . . 
+ Zi2^ -H Zo2°. A next height position hp(z) is determined using a process 
including three steps. First, binary coefficients starting with coefficient Zq, up to 
and but not including coefficient Zj- are reversed in order while coefficients and 

25 above are kept the same. Thus, after the first step the height position becomes Zj. 
{2}-^ + Zj.22^-2 + . . . + Zr2T + zo20 + . . . + t^,^^-'^ + 

Second, an odd number modulus i} , for example one, is added to the height 
position after inversion. Third, circularity of the height position is enforced by 
limiting the inverted and incremented height position by modulus 1^. Fourth, the 

30 first step is repeated, again inverting the binary coefficients below the z^ 
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coefficient of the previously inverted, incremented and limited height position. 
The inverse operation for deriving height descriptor Hx(z) is determined in the 
same manner except that, rather than adding the odd niunber modulus 2"^ to the 
order-inverted bit string, die same odd number modulus 2"^ is added to the order- 
5 inverted bit string. 

The interconnection between nodes 102 on the same level is notable and 
highly advantageous for many reasons. For example, the interconnection structure 
resolves contention among messages directed toward the same node. Also, the 
intercoimection structure ensures that a message on a particular level that is 

10 blocked by messages on the next level makes a complete tour of the messages on 
that level before any message is in position to block again. Thus a message 
inherently moves to cover all possible paths to the next level. Furthermore, a 
blocking message must cycle through all rings of a level to block a message twice. 
Consequently, every message is diverted to avoid continuously blocking other 

15 messages. In addition, blocking messages typically proceed to subsequent levels 
so that overlymg messages are not blocked for long. 

When messages are sent from second data output terminal 222 of a node 
N(T,e,z) 450 to a second data input terminal 212 of a node N(T,e+l,hT-(z)), a 
control code is also sent from a control output terminal 224 of the node N(T,0,z) 

20 450 to a control input terminal 214 of a node N(T+ 1 ,0,hT+i(z)), the node on level 
T+1 that has a data output terminal connected to a data input terminal of node 
N(T,0 + l,hT.(z)). This control code prohibits node N(T+l,0,hx+i(z)) from 
sending a message to node N(T,04-l,hT+i(z)) at the time node N(T,e,z) 450 is 
sendmg a message to node N(T,^-f-l,hx+i(z)). When node N(T+l,5,h-r+i(z)) is 

25 blocked from sending a message to node N(T,^+l,hr+i(z)), the message is 
deflected to a node on level T+1. Thus, messages communicated on the same 
level have priority over messages communicated from another level. 
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The second data output terminal 222 of nodes N(T,^-1,Ht(z)) are connected 
to a second data input terminal 212 of nodes N(T,0,z) 450 so that nodes N(T,0,z) 
450 receive messages from nodes N(T,^-1,Ht(z)) that are blocked from 
transmission to nodes N(T-l,0,Hi-(z)). Also, the control output terminal 224 of 
5 nodes N(T-1,^,Ht{z)) to the control input terminal 214 of nodes N(T,e,z) 450 to 
warn of a blocked node and to inform nodes N(T,0,z) 450 not to send data at this 
time since no node receives data from two sources at the same time. 

Referring to Figure 13, interconnections of nodes 102 on cylindrical level 
one exemplify the described interconnections and demonstrate characteristics and 

10 advantages that arise from the general interconnection t^hnique. In this example, 
the number of nodes K at a cylindrical height is five and the number of heights 2^ 
is 2^, or 4, for a three level (J 4- 1) mterconnect structure 500. Nodes N(1.0,z) 510 
have: (1) a first data input termmal 210 connected to a first data output terminal 
220 of nodes N(2,e-l,z) 512, (2) a control output termmal 224 connected to 

15 control input termmal 214 of nodes N(2,^,h2(z)) 512, (3) a first data output 
terminal 220 connected to a first data input terminal 210 of nodes N(O,0+l,z) 516, 
(4) a control input terminal 214 connected to a control output terminal 224 of 
nodes N(0,^,Hr(z)) 516, (5) a second data mput termmal 212 connected to the 
second data output terminal 222 of nodes N(I,e-l,Hi(z)) 520, and (6) a second 

20 data output terminal 222 connected to the second data input termmal 212 of nodes 
N(l,^+l,hi(z)) 522. For nodes N(I,^,z) 510 on level one, height z differs from 
height hi(z) and height Hi(z) only in the final bit position. 

Messages are communicated through the mtercoimect structure 500 in 
discrete time steps. A global clock (not shown) generates timing signals in 
25 discrete time steps modulus the number of nodes K at a cylindrical height z of a 
cylindrical level r. When messages traverse the interconnect structure 500 on the 
same level (for example, level one) because nodes on an inner level are blocked, 
messages are communicated from node to node in the discrete time steps. For the 
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interconnect structure 500 with an odd number (K=5) of nodes at a cylindrical 
level, if data traverses level one for 2K time steps, then the message packet visits 
2K different nodes. On time step 2K+1, message packets will begin repeatmg 
nodes following the sequential order of the first node traversal. Because the global 
5 clock generates the discrete tune steps integral time modulus K, if a message 
packet on level one is over the target ring of that packet at a time T=0 (modulus 
K) and is deflected by a message on level zero, the message will be over the target 
ring also at a time T=0 (modulus K) to make another attempt to enter the target 
ring. In various embodiments, this timing characteristic is consistent throughout 
10 the interconnect structure so that, if a message packet is in a position to descend 
to the next level at a time T=0 (modulus K), the packet will once again be in a 
position to descend at a subsequent time T=0 (modulus K). 

Referring to Figure 14 in conjunction with Figure 13, interconnections of 
nodes 102 on cylmdrical level two further exemplify described interconnections. 

15 In Figure 14, a level two message path 620 is shown overlying the paths 610 and 
612 of messages moving on level one. The number of nodes K at a cylindrical 
level is five and the number of levels 2^ is 2^, or 4, for a three level (J+1) 
interconnect structure 500. Same-level interconnections of nodes N(2,^,z) include: 
(1) a second data input terminal 212 connected to the second data output terminal 

20 222 of nodes N(2,^-l,h2(z)) and (2) a second data output tennmal 222 connected 
to the second data input tennmal 212 of nodes N(2,^+l,H2(z)). For nodes 
N(2,0,z) on level two, height z differs from height h2(z) and height H2(z) only in 
the final two bit positions. Generally stated in binary form for any suitable 
number of nodes K at a height and number of heights 2-' in a level, bits z and h2(z) 

25 on cylindrical level two are related as follows: 

[Zj.i, Zj.2, . . Z2, 0, 0]' = [Zj.j, Zj.2, . . ., Zj. 1, 0]; 

[zj.i, Zj.2 Z2, 1, 0]' = [zj.i, Zj.2, . . ., Z2, 0, 1]; 

[zj.j, Zj.2, . . •, Z2, 0, 1]' = [zj.i, Zj.2, . . Zj, 1, 1]; and 

[Zj.i, Zj.2, . . Z2, 1, 1]' = [Zj.i, Zj.2, . . Z2, 0, 0]. 
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A second advantage of this interconnection technique for nodes on the same 
level is that blocked messages are directed to avoid subsequent blocking. Figure 
14 illustrates a message blocking condition and its resolution. On level one, a 
message mo 610 is shown at node Nqoi and a message 612 at node Nqh. A 
5 message M 620 on level two at node Nqoz is targeted for ring zero. At a time 
zero, message M 620 is blocked and deflected by message mj 612 to node N122 
at time one. Assummg that messages hiq and mi are also deflected and traversing 
level one, at a time one message hiq 610 is at node Nm and message m^ 612 at 
node Nioi- At a tune two, message M 620 moves to node N212, message hiq 610 
10 to node N201 and message m^ 612 to node N211. Thus, at tune two, message M 
620 is deflected by message toq 610. At tune four, message M 620 is again 
blocked by message m^ 612. This alternating blocking of message M 620 by 
messages m^, 610 and 612 contmues indefinitely as long as messages hiq 610 
and mi 612 are also blocked. This characteristic is pervasive throughout the 
15 interconnect structure so that a single message on an inner level cannot contmue 
to block a message on an outer level. Because a single message packet cannot 
block another packet and blockmg packets contmually proceed through the levels, 
blocking does not persist. 

Referring to Figure 15, interconnections of nodes 102 on cylindrical level 
20 three show additional examples of previously described interconnections. A level 
three message path 720 is shown overlying the patiis 710, 712 and 714 of 
messages moving on level two. The number of nodes K at a cylindrical height is 
seven and the number of heights 2^ is 2^ (8), for a four level (J+1) mterconnect 
structure. Same-level interconnections of nodes N(3,^,z) include:- (1) a second 
25 data input terminal 212 connected to the second data output terminal 222 of nodes 
N(3,0-l,h3(z)) and (2) a second data output terminal 222 connected to the second 
data input terminal 212 of nodes N(3,^ + l,H3(z)). For nodes N(3,0,z) on level 
three, height z differs from height h^iz) and height HjCz) only m the final three bit 
positions. Generally stated in binary form for any suitable number of nodes K at 
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a cylindrical height and number of heights l) in a level, bits z and hjCz) on 



cylindrical level three are related as 


follows: 
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Figure 15 illustrates another example of a message blocking condition and 
its resolution. On level two, a message uiq 710 is shown at node Noo2> ^ message 
m^ 712 at node N012, a message va-i 714 at node Nq22 and a message m3 716 at 
node No32. A message M 720 on level three at node N303 is targeted for ring zero. 

15 At a time zero, message M 720 is blocked and deflected by message 716 to 
node N173 at time one. Assummg that messages mo, mj, m2 and m3 are also 
deflected and traversing level two, at a time one message hiq 710 is at node Ni32, 
message mj 712 at node Ni22> message m2 714 at node N102 and message m3 716 
at node N^j- At a time two, message M 720 moves to node N233, message mo 

20 710 to node N212, message m^ 712 to node N202. message m2 714 to node N232 and 
message m3 716 to node N222- Thus, at time two, message M 720 is deflected by 
message 712. At time four, message M 720 is blocked by message m2 714. 
At time six, message M 720 is blocked by message mo 710. At time eight, 
message M 720 is again blocked by message m3 716. This alternating blocking 

25 of message M 720 by messages nio 710, mi 712, mj 714 and 716 continues 
indefinitely as long as messages mo 710, m^ 712, m2 714 and m3 716 are also 
blocked. 

This analysis illustrates the facility by which the described interconnect 
structure avoids blocking at any level. Thus, "hot spots" of congestion in the 
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structure are minimized. This characteristic is maintained at all levels in the 
structure. 

The described interconnect structure provides that every node N(0,6,z) on 
level zero is accessible by any node N(J,^,z) on outermost level J. However, only 
5 half of the nodes N(0,^,z) on level zero are accessible by a node N(J-l,6,z) on the 
level once removed from the outermost level. Data at a node N(l,6,z) on level 
one can access any node N{0,6,z) on level zero so long as the binary 
representation of height z of level one and the bmaiy representation of ring r of 
level zero differ only in the last bit. Similarly, data at a node N(2,d,z) on level 

10 two can access any node N(0,$,z) on level zero so long as the binary 
representation of height z of level two and the binary representation of ring r of 
level zero differ only in the last two bits. A general rule is that, data at a node 
N(T,6,z) on level T can access any node N(0,$,z) on level zero so long as the 
binary representation of height z of level T and the binary representation of ring 

15 r of level zero differ only in the last T bits. Accordingly, moving from the 
outermost level J to level J-1 fixes the most significant bit of the address of the 
target ring. Moving from level J-1 to level J-2 fixes the next most significant bit 
of the address of the target ring and so forth. At level zero, no bits are left to be 
fixed so that no header bit is tested and a message is always passed to a device. 

20 In some embodiments, an additional header bit is included and tested at a level 
zero node. This final bit may be used for various purposes, such as for directing 
message data to a particular buffer of a device when the device accepts the 
message data. An advantage of including an additional bit in the header and 
performing a bit test at the final node is that all the nodes at all levels of the 

25 interconnect structure operate consistently. 



In some embodiments of an interconnect structure, an additional header bit 
is included in a message packet. This bit indicates that a message packet is being 
transmitted. Another purpose for such an additional bit in the header is to identify 
which bit in the header is the control bit. 
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A message packet moves from a level T to the next imier level T-1 so long 
as two conditions are met, as follows: (1) the target ring of the message packet is 
accessible from level T-1, and (2) the message packet is not blocked by a message 
on the level T-1 . 

5 One significant aspect of this structure is that any message packet at a node 

N(T,^,z) on a level T that can access its target ring can also access the target ring 
from a node N(T-l,6 + l,z) only if the bit T-1 of the address ring is the same as 
bit T-1 of the target ring. Therefore, analysis of only a single bit yields all 
information for determining a correct routing decision. 

10 Referring to Figure 16, a general relationship between message packets on 

two adjacent levels T and T+1 is described. In this example, a message packet 
M at a node N45Q on level four, which is targeted for ring zero, is potentially 
blocked by eight message packets uiq 810, m^ 811, m2 812, 813, m4 814, 
815, m^ 816 and m7 817 at nodes N3}o residing on each of the heights 0 to 7 on 

15 level three. Although the behavior of the interconnect structure is analyzed with 
respect to levels three and four for purposes of illustration, the analysis is 
applicable to any arbitrary adjacent levels. At an arbitrary time step, illustratively 
called time step zero, the message M moves from node N450 on level four to node 
N351 on level three unless a control code is send to node N450 from a level three 

20 node havmg a data output terminal connected to node N351. In this example, node 
N310 has a data output terminal connected to node N351 and, at time step zero, 
message m-j 817 resides at node N310. Accordingly, node N310 sends a control 
code, in this example a single bit code, to node N450, causing deflection of 
message M to node N^qi (where D is a hexadecunal designation of 13) on an 

25 interconnection line. A bit line illustratively shows the control coimection from 
node N310 to node N450. At a time step one, message M moves from node N401 
to node N432 on interconnection line regardless of whether a node N328 is blocked 
because ring zero is not accessible from node N328. At a time step two, message 
M moves from node N432 to node N334 unless a control blocking code is sent from 
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node N352 to node N432 where node N352 is the node on level three that has a data 
output terminal connected to a data input terminal of node N334. However, the 
message M is blocked from accessing node N334 because message m^ currently 
resides at node N352 at time step two. A deflection control code is sent from node 
5 N352 to node N432 on control bit line 822. Furthermore, assuming that none of the 
message packets mj progresses to level two and beyond, at time step four, message 
M is blocked by message m2 via a control code sent on control bit line. At time 
six, message M is blocked by message m4 though a blocking control code on 
control bit line. 

10 This example illustrates various advantages of the disclosed interconnection 

structure. First, deflections of the message M completely tour all of the heights 
on a level T if messages mj on level T-1 continue to block progression to the level 
T-1 for all levels T. Accordingly, a message M on a level T is blocked for a 
complete tour of the heights only if 2^'^ messages are in position on level T-1 to 

15 block message M. In general, a message mj on a level T-1 must remain on the 
level T-1 for 2'^'^^ time steps to block the same message M on level T twice. 

The description exemplifies an interconnect structure in which messages 
descend from an outer level to devices at a core inner layer by advancing one level 
when the height dimension matches the destination ring location and traversing the 

20 rings when the ring location does not match the height designation. In other 
embodiments, the messages may move from an inner level to an outer level. In 
some embodiments, the heights may be traversed as the level changes and the 
height held constant as the level remains stationary. In these embodiments, the 
progression of messages through nodes is substantially equivalent to the disclosed 

25 intercoimect structure. However, the advantage of the disclosed network that 
avoids blocking of messages is negated. 

Referring to Figure 17, a timing diagram illustrates timing of message 
communication in the described interconnect structure. In various embodiments 
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of the interconnect structure, control of message communication is determined by 
timing of message arrival at a node. A message packet, such as a packet 900 
shown in Figure 18, includes a header 910 and a payload 920. The header 910 
includes a series of bits 912 designating the target ring in a binary form. When 
5 a source device CU(0i,Zi) at an angle 6^ and height sends a message packet M 
to a destination device CU(d2,7^) at an angle 62 and height Z2, the bits 912 of 
header 910 are set to the binary representation of height Zj. 

A global clock servicing an entire interconnect structure keeps integral time 
modulus K where, again, K designates the number of nodes n at a cylinder height 

10 z. There are two constants a and |3 such that the duration of a exceeds tiie 
duration of /S and the following five conditions are met. First, the amount of time 
for a message M to exit a node N(T,e + l,hx(z)) on level T after exiting a node 
N(T,0,z) also on level T is a. Second, the amount of time for a message M to 
exit a node N(T-l,^+l,z) on level T-1 after exiting a node N(T,^,z) on level T is 

15 a - 18. Third, the amount of time for a message to travel from a device CU to a 
node N(T,e,z) is a - /3. Fourth, when a message M moves from a node N(r,0,z) 
to a node N(r,0 + l,hr(z)) in time duration a, the message M also causes a control 
code to be sent from node N(r,0,z) to a node N(r+1,0 + I,hr(z)) to deflect 
messages on the outer level r+1. The tune that elapses from the time that 

20 message M enters node N(r,0,z) until the control bit arrives at node 
N(r -h 1 ,0 + 1 ,\+i(z)) is time duration jS . The aforementioned fourth condition also 
is applicable when a message M moves from a node N(J,e,z) to a node 
N(J,0+l,hj(z)) at the outermost level J so that the message M also causes a control 
code to be sent from node N(J,^,z) to a device CU(e,z). The time, that elapses 

25 from the time diat message M enters node N(r,0.z) until the control bit arrives at 
device €13(6, z) is tune duration jS. Fifth, the global clock generates timing pulses 
at a rate of a. 

When the source device CU(0i,Zi) sends a message packet M to the 
destination device CUie2,z^, the message packet M is sent from a data ou^ut 



M-3240 PCT 

-33- 

terminal of device CUC^i.Zi) to a data input tenninal of node N(J,^i,Zi) at the 
outennost level J. Message packets and control bits enter nodes N(T,e,z) on a 
level T at times having the form na + L/3 where n is a positive integer. The 
message M from device CU(,d^,Zi) is sent to the data mput terminal of node 
5 N(J,^i,Zi) at a time to - i5 and is inserted into the data input terminal of node 
N(J,ei,Zi) at time to so long as the node N(J,^i,Zi) is not blocked by a control bit 
resulting from a message traversuig on the level J. Time to has the form (Sj-^i)©: 
+J/3. Similarly, there is a time of the form (^2-^i)a +W at which a data input 
terminal of node N(J,0i,Zi) is receptive to a message packet from device 
10 CU((9i,zi). 

Nodes N(r,0,z) include logic that controls routing of messages based on the 
target address of a message packet M and timing signals from other nodes. A first 
logic switch (not shown) of node N(r,0,z) determines whether the message packet 
M is to proceed to a node N(T-l,0-f- l,z) on the next level T-1 or whether the node 

15 N(T-l,e + 1 ,z) is blocked. The first logic switch of node N(r,0,z) is set according 
to whether a single-bit blocking control code sent from node N(T-1,^,Ht(z)) 
arrives at node N{t,6,z) at a time to- For example, in some embodiments the first 
logic switch takes a logic 1 value when a node N(T-l,^+l,z) is blocked and a 
logic 0 value otherwise. A second logic switch (not shown) of node N(r,5,z) 

20 determines whether the message packet M is to proceed to a node N(T-l,0+l,z) 
on the next level T-1 or whether the node N(T-l,0+l,z) is not in a suitable path 
for accessmg the destmation device 0X5(62,2^) of the message packet M. The 
message packet M includes the binary representation of destination height Z2 (Ta^y 
Z2(j.i), . . , Z2(j), . . , Z2(i), Z2(0). The node N(T,0,z) on level T mcludes a single- 

25 bit designation 2>j- of the height designation z (zj, Zj.i, . . , Zp, . . , z^, z^). In this 
embodiment, when the first logic switch has a logic 0 value and the bit designation 
Z2(j) of the destination height is equal to the height designation 7^, then the 
message packet M proceeds to the next level at node N(T-l,0+l,z) and the 
destination height bit Z2(t) is stripped from the header of message packet M. 

30 Odierwise, the message packet M traverses on the same level T to node 
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N(T,0 + l,hT(z)). If message packet M proceeds to node N(T-l,^+l,z), then 
message packet M arrives at a time tg + (a - j3) which is equal to a time ^ - Zj 
+ l)a + (J - If message packet M traverses to node N(T,e+l,hT-(z)), then 
message packet M arrives at a time t^ + a, which is equal to a time (Zj - Zi + 
5 \)a + J/3. As message packet M is sent from node N(r,0,z) to node 
N(T,0+l,h-r(z)), a single-bit control code is sent to node N(T+1,^+1,Ht+i(z)) 
(or device CU(0,z) which arrives at time to + /3. This timing scheme is continued 
diroughout the interconnect strucmre, maintaining synchrony as message packets 
are advanced and deflected. 

10 The message packet M reaches level zero at the designated destination 

height Z2. Furthermore, the message packet M reaches the targeted destination 
device CU(02'^) at a time zero modulus K (the number of nodes at a height z). 
If the targeted destmation device CU(02'22) is ready to accept the message packet 
M, an input port is activated at time zero modulus K to accept the packet. 

1 5 Advantageously , all routing control operations are achieved by comparing two bits , 
without ever comparing two multiple-bit values. Further advantageously, at the 
exit point of the interconnect strucmre as message packets proceed from the nodes 
to the devices, there is no comparison logic. If a device is prepared to accept a 
message, the message enters the device via a clock-controlled gate. 

20 Many advantages arise as a consequence of the disclosed timing and 

interconnect scheme. In an optical implementation, rather than an electronic 
implementation, of the interconnect structure, signals that encode bits of the header 
typical have a longer duration than bits that encode the payload. Header bits are 
extended in duration because, as messages communicate through the interconnect 

25 structure, timing becomes slightly skewed. Longer duration header bits allow for 
accurate reading of the bits even when the message is skewed. In contrast, 
payload bits encode data that is not read during communication through the 
interconnect structure. The disclosed timing scheme is advantageous because the 
number of header bits' in a message is greatly reduced. Furthermore, in some 
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embodiments the number of header bits is decremented as bits are used for control 
purposes at each level then discarded while messages pass from level to level in 
the interconnect structure. In embodiments that discard a control bit for each level 
of the interconnect structure, logic at each node is simplified since the control bit 
5 at each level is located at the same position throughout the interconnect structure. 

That messages communicated on the same level have priority over messages 
communicated from another level is similarly advantageous because message 
contention is resolved without carrying priority information in the message header. 
Message contention is otherwise typically resolved by giving priority to messages 
10 that have been in an interconnect structure the longest or to predetermined 
prioritization. These techniques use information stored in the header to resolve 
contention. 

Although it is advantageous that the interconnect structure and message 
communication method determines message transmission routing using self-routing 

15 decision-making which is local to the nodes and depends on message timing, in 
some embodiments of the control structure, both local and global communication 
control is employed. For example, one embodiment of an interconnect structure 
uses local control which is based on timing to control transmission of message 
packets in a first transmission mode and alternatively uses global control via a 

20 scheduler to administer communication of lengthy strings of message data m a 
second mode. In the global mode, the usage of a scheduler makes the usage of 
control bit input and output terminals uimecessary. 

One consequence of self-routing of message packets is that the ordering of 
message packet receipt at a target device may be variable. In some embodiments, 
25 the correct order of message segments is ordered by sending ordering information 
in the message header. Other embodiments employ an optical sorter to order 
message packets. 
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Although many advantages are realized through a control structure and 
communication method which utilizes timing characteristics, rather than control 
bits in the header, to control message routing, some interconnect node technologies 
more suitably operate in a routing system utilizing no timing component. Thus in 
5 these technologies, instead of introducing a message at predetermined time so that 
the message arrives at a preset destination at a designated, routing information is 
contained in additional header bits. Accordingly, a designated target device 
position is included in header bits, for example bits followmg the designated target 
ring position. 

10 In one embodiment, the label of a target device is represented as a single 

logic one in a string of logic zeros. Thus, when a message arrives at a device N, 
the device samples the Nth bit of the device element of the header (as distinguished 
from die ring element) and accepts the message if the Nth bit is a logic one. This 
technique is highly suitable for optical node implementations. 

15 

Nodes 

The nodes N(r,^,z) have been described in generic terms to refer to various 
data commimication switches for directing data to alternative data paths. Node 
structures which are presently available include electronic nodes, optical nodes and 

20 mixed optical/electronic nodes. What is claimed include, for example, 
interconnect and timing methods, an interconnect apparatus and an intercoimect 
topology. These methods and apparati involve nodes in a generic sense. Thus, 
the scope of the claims is not limited by the particular type of node described 
herein and is to extend to any node known now or in the future, which performs 

25 the function of the nodes described herein. 

One example of a node 1300 is shown, referring to Figure 19, which 
includes a lithium niobate (LiNb03) gate 1302. The lithium niobate gate 1302 has 
two data input terminals 1310 and 1312, two data output terminals 1320 and 1322 
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and one control input terminal 1330. Various control circuitry 1340 is added to 
the lithium niobate gate 1302 to form a control output terminal 1332 of the node 
1300. Node 1300 also includes optical to electronic converters 1354, 1356 and 
1358. The lithium niobate gate 1302 is forms a 2x2 crossbar. Data paths 1342 
5 and 1344 are optical and the control of the node 1300 is electronic. The lithium 
niobate gate 1302 is combined with a photodetectors 1350 and 1352 and a few 
electronic logic components to form a node 1300 for various embodiments of an 
interconnect structure. 

In operation, as a message packet 1360 approaches the node 1300, part of 
10 the message packet signal 1360 is split off and an appropriate bit of the message 
packet header (not shown) designating a bit of the binary representation of 
destination ring in accordance with the discussion hereinbefore, is read by the 
photodetector 1350. This bit is converted from optical form to an electronic 
signal. This bit, a bit designating the cylinder height upon which the node 1300 
15 lies and a bit designating whether a destination node on the next level is blocked 
are processed electronically and a result of the logical tests of these bits is directed 
to the control mput terminal 1330 of the lithium niobate gate 1302. In a first type 
of lithium niobate gate technology, if the result signal is a logic zero, the gate 
switches in the cross state. In a second type of lithium niobate gate technology, 
20 a logic zero result signal switches the gate in a bar (straight through) state. 

Referrmg to Figure 20, an additional example of a node 1400 is shown. 
Node 1400 uses a nonlinear optical loop mirror (NOLM) 1410 to perform a 
switching fimction. A nonlinear optical loop mirror is a device that makes use of 
the refractive index of a material to form a completely optical switch that is 
25 extremely fast. One example of a NOLM switch includes a data input terminal 
1412 and a control mput terminal 1414. Depending upon the signal at the control 
input termmal 1414, data either leaves the NOLM 1410 through the same data 
input termmal 1412 from which the data entered (hence the term mirror) or the 
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data exits through a data output terminal 1416. Data is polarized and split into two 
signal "halves" of equal intensity. In the absence of a control pulse, the two 
halves of the signal recombine and leave the NOLM 1410 through die data input 
terminal 1414. When a control pulse is applied to the control input terminal 1414, 
5 the control pulse is polarized at right angles to the data pulse and inserted into the 
NOLM 1410 so that the control pulse travels with one half of the data pulse. The 
control pulse is more intense than the data pulse and the combined first half of the 
data pulse and the control pulse quickly pass the second half of the data pulse so 
that the second half of the data pulse is only minimally accelerated. Thus, the two 
10 halves of the data pulse travel with slightly different velocities and are 180° out 
of phase when the two halves are recombined. This phase difference causes the 
combined data pulse signal to pass through the data output terminal 1416. One 
disadvantage of the NOLM 1410 is that switching is operational only when a long 
optical transmission loop is employed, thus latency is a problem. 

15 Referring to Figure 21, another example of a node 1500 is shown which 

uses a terahertz optical asymmetrical demultiplexer (TOAD) switch 1510. The 
TOAD switch 1510 is a variation of the NOLM switch 1410. The TOAD 1510 
includes an optical fiber loop 1512 and a semiconductor element 1514, a nonlinear 
element (NLE) or a semiconductor optical amplifier for example. The TOAD 

20 switch 1510 has an mput data terminal 1520 which also serves as an output data 
port under some conditions. The TOAD switch 1510 also has a separate second 
output data terminal 1522. The semiconductor element 1514 is placed 
asymmetrically with respect to the center 1516 of the fiber optic loop 1512. A 
distance 1518 from the semiconductor element 1514 to the center 1516 of the fiber 

25 optic loop 1512 is the distance to transmit one bit of data. The TOAD 1510 
functions by removing a single bit from a signal having a high data rate. The 
TOAD 1510 is switched by passing a constant electrical current through die 
semiconductor element 1514. An optical signal entering the semiconductor 
material causes the index of refraction of the material to immediately change. 
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After the optical signal terminates, the mdex of refraction slowly (a time span of 
several bits) drifts back to the level previous to application of the optical signal. 
A control pulse is an optical signal having an intensity higher than that of an 
optical data signal and polarization at right angles to the optical data signal. An 
5 optical data input signal is polarized and split into two signal "halves" of equal 
intensity. The control pulse is injected in a manner to move through the fiber 
optic loop 1512 directly over the bit that is to be removed. Because the distance 
1518 is exactly one bit long, one half of the split optical data signal corresponding 
to a bit leaves the semiconductor element 1514 just as the other half of the bit 

10 enters the semiconductor element 1514. The control pulse only combines with one 
half of the optical data signal bit so that the velocity of the two halves differs. The 
combined data and control signal bit exits the TOAD 1510 at the input data 
terminal 1520. Thus, this first bit is removed from the data path. A next optical 
signal bit is split and a first half and second half, moving in opposite directions, 

15 are delayed approximately the same amount as the index of refraction of the 
semiconductor element 1514 gradually changes so that this bit is not removed. 
After a few bits have passes through the semiconductor element 1514, the 
semiconductor material relaxes and another bit is ready to be multiplexed fi-om the 
optical data signal. Advantageously, the TOAD 1510 has a very short optical 

20 transmission loop 1512. 

Regenerators 

It is a characteristic of certain nodes that messages lose strength and pick 
up noise as they propagate through the nodes. Using various other nodes, message 
signals do not lose strength but noise accumulates during message transmission. 
25 Accordingly, in various embodiments of the interconnect structure, signal 
regenerators or amplifiers are used to improve message signal fidelity after 
messages have passed through a number of nodes. 
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Referring to Figure 22, one embodiment of a regenerator 1600 is shown 
which is constructed using a lithium niobate gate 1602. A lithium niobate gate 
1602 regenerates message data having a transmission speed of the order of 2.5 
gigabits. The lithium niobate gate 1602 detects and converts an optical message 
5 signal to an electronic signal which drives an electronic input port 1604 of the 
lithium niobate gate 1602. The lithium niobate gate 1602 is clocked using a clock 
signal which is applied to one of two optical data ports 1606 and 1608 of the gate 
1602. The clock signal is switched by the electronic control pulses and a high 
fidelity regenerated signal is emitted from the lithium niobate gate 1602. 

10 Typically, an interconnect strucmre utilizmg a lidiium niobate gate 1602 in 

a regenerator 1600 also uses lithium niobate gates to construct nodes. One large 
power laser (not shown) supplies high fidelity timuig pulses to all of the 
regenerators in an interconnect strucmre. The illustrative regenerator 1600 and 
node 1650 combination mcludes an optical coupler 1620 which has a first data 

15 input connection to a node on the same level C as the node 1650 and a second data 
input connection to a node on the overlying level C+1. The illustrative 
regenerator 1600 also includes a photodetector 1622 connected to an output 
terminal of the optical coupler 1620, optical to electronic converter 1624 which has 
an input terminal connected to the optical coupler 1620 through the photodetector 

20 1622 and an output terminal which is connected to the lithium niobate gate 1602. 
An output terminal of the lithium niobate gate 1602 is connected to a second 
lithium niobate gate (not shown) of a node (not shown). Two signal lines of the 
lithium niobate gate (not shown) are combined, regenerated and switched. 

When regenerators or amplifiers are incorporated to improve signal fidelity 
25 and if the time expended by a regenerator or amplifier to recondition a message 
signal exceeds the time a - /3, dien the regenerator or amplifier is placed prior to 
the input terminal of the node and tuning is modified to accommodate the delay. 
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Other Embodiments 

The interconnect structure shown in Figures 1 through 16 is a simplified 
structure, meant to easily convey understanding of the principles of the invention. 
Numerous variations to the basic structure are possible. Various examples of 
5 alternative interconnect strucmres are discussed hereinafter, along with advantages 
achieved by these alternative structures. 

Referring to Figure 23, an alternative embodiment of an iuterconnect 
structure 1000 includes devices 1030 which issue message packets to multiple 
nodes 1002 of the outermost level J. In the interconnect apparatus 100 shown in 

10 Figures 1 through 16, a device CU(0,z) initiates a message transmission operation 
by sending a message packet to a node N(J,5,z). In the alternative interconnect 
structure 1000, the device CU(0,z) initiates a message transmission operation by 
sending a message packet to node N(J,0,z) but, in addition, also includes 
interconnections to additional multiple nodes N(J,0,z) where z designates cylinder 

15 heights selected from heights 0 to 2^ of the outermost level J and 0 designates node 
angles selected from angles 0 to K of the heights z. In the case that a device sends 
messages to more than one node in the outermost level, the disclosed timing 
scheme maintains the characteristic that messages arrive at the target node at time 
zero modulus K. 

20 Devices are connected to many nodes in the outermost level J to avoid 

congestion upon entry into the intercoimect strucmre caused by multiple devices 
sending a series of messages at a high rate to nodes having converging data paths. 
In some embodunents, the nodes to which a device is connected are selected at 
random. In other embodiments, the multiple interconnection of a device to several 

25 nodes is selected in a predetermined manner. An additional advantage arising 
from the coimection of a device to several nodes increases the input bandwidth of 
a communication network. 
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Referring to Figure 24, an alternative embodiment of an interconnect 
structure 1100 includes devices 1130 which receive message packets from multiple 
nodes 1102 of the innermost level 0. In this example, the number of nodes K at 
a particular height z is nine and each device 1130 is connected to receive message 
5 from three nodes on level zero. The interconnect structure 1100 is advantageous 
for improving network exit bandwidth when the number of nodes K on at a 
particular height is large. 

In the example in which the number of nodes K on a height z is nine and 
each device receives messages from three nodes on level zero, each node on ring 

10 zero is connected to a buffer that has three levels. At time 0, message data is 
injected into the level zero buffer. At time three, data is injected into the level one 
buffer. At time 6, data is injected into the level two buffer. A device CU(^,0) 
reads from the level zero buffer at node N(0,6,0), from the level one buffer at 
node N(0,(^+3)mod 9,0), and from the level two buffer at node N(O,(0+6)mod 

15 9,0). This reading of message data is accomplished in a synchronous or 
nonsynchronous manner. If in the synchronous mode, a time t is expended to 
transfer data from the buffer to the device. In this case, the device CU(^,0) reads 
from the level zero buffer at time t, reads from the level three buffer at time 3+t, 
and reads from the level six buffer at time 6+t. In an asynchronous mode, device 

20 CU(0,O) interconnects to the three buffers as described hereinbefore and reads 
message data whenever a buffer signals that data is available. 

Referring to Figure 25, an alternative embodiment of an interconnect 
structure 1200 includes devices 1230 which issue message packets to multiple 
nodes 1202, not only in the outermost level J but also in other levels. In the 
25 alternative interconnect strucmre 1200, the device CU{d,z) initiates a message 
transmission operation by sending a message packet to node N(J,^,z) but, in 
addition, also includes interconnections to additional multiple nodes N(T,6,z) where 
r designates levels of the interconnect structure 1200, z designates cylmder heights 
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selected from heights 0 to 2^ of the outermost level J and 6 designates node angles 
selected from angles 0 to K of the heights z. In the case that a device sends 
messages to nodes in more than level, message communication is controlled 
according to a priority, as follows. First, messages entering a node N(r,^,z) from 
5 the same level T have a first priority. Second, messages entering a node N(r,0,z) 
from a higher level T+1 have a second priority. Messages entering a node 
N(r,0,z) from a device CU(0,z) have last priority. The alternative embodiment of 
interconnect structure 1200 allows a device to send messages to neighboring 
devices more rapidly. The disclosed timing scheme maintains the characteristic 
10 that messages arrive at the node designated in the message header at time zero 
modulus K. 

In these various embodiments, devices accept data from level zero nodes 
using one of various predetermined techniques. Some embodiments rely 
exclusively on timing to determine when the devices accept data so that devices 
15 accept data at time zero modulus K. Some embodiments include devices that 
accept message data at various predetermined times with respect to modulus K 
timmg. Still other embodiments have devices that accept data whenever a buffer 
is ready to accept data. 

. Wave Division Multiplexing Embodiment 

20 In another embodiment of an interconnect structure, message signal 

bandwidth is increased using wave division multiplexing. A plurality of K colors 
are defined and generated in a message signal that is transmitted using an 
interconnect structure having K devices at a cylinder height. Accordingly, each 
device is assigned a particular color. Message packets travel to a preselected 

25 target ring in the manner described hereinbefore for a single wavelength 
mterconnect system. Message packets pass from level zero to the appropriate 
device depending on the color assigned to the message packet. 
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A message includes a header and a pay load. The header and pay load are 
distinguished by having different colors. Similarly, the pay load is multiplexed 
using different colors, which are also different from the color of the header. 
Message bandwidth is also increased by combining different messages of different 
5 colors for simultaneous transmission of the messages. Furthermore, different 
messages of different colors are bound on a same target ring, combined an 
transmitted simultaneously. All messages are not demultiplexed at all of the nodes 
but, rather, are demultiplexed at input buffers to the devices. 

Variable Base f Height Structure Embodiment 

10 

In a ftirther additional embodiment, an interconnect structure has i^ 
cylindrical heights on a level for each of J+1 levels, where i is a suitable integer 
number such as 2 (the previously described embodiment), 3, 4 or more. As was 
described previously, each height contains K nodes, and each node has two data 
15 input terminals, two data output terminals, one control mput terminal and one 
control output terminal. 

For example, an interconnect structure may have 3^ heights per level. On 
level one, message data is communicated to one of three level zero heights. On 
level two, message data is communicated to one of nine level zero heights and so 

20 forth. This result is achieved as follows. First, the two output data termmals of 
a node N(r,^,z) are connected to input data termmals of a node N(T-l,^+l,z) and 
a node N(T,0+l,hT(z)), in the manner previously discussed. However in this 
further embodiment, a third height transformation hj(hx(br(z))) rather than a 
second height transformation hjCh-rCz)) is equal to the original height designation 

25 z. With the nodes mterconnected in this manner, the target ring is accessible to 
message data on every third step on level one. In an interconnect strucmre having 
this form, although nodes having two output data terminals are suitable, advantages 
are gained by increasing the number of output data terminals to three. Thus, one 
data output terminal of a node on a given level is connected to two nodes on that 
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level and to one node on a successive level. Accordingly, each level has 3^ heights 
and a message packet and a message can descend to a lower level every other step. 

In this manner, many different interconnect structures are formed by 
utilizing i^ heights per level for various numbers i. Where i is equal to 4, the 
5 fourth height transformation hi-(h-p(hT(hx(z)))) is equal to the original height 
designation z. If i is 5, the fifth height transformation hr(hx(h-i-(bj-(hT(z))))) is die 
same as the original height z, and so forth. 

In the variable base i^ height structure embodiment, whether the target ring 
is accessible from a particular node is determined by testing a more than one bit 
10 of a code designating the target ring. 

Variable Base Transformation Technique Embodiment 

In a still further embodiment of an interconnect structure, the height 
transformation technique outlined hereinbefore is modified as follows. In this 
embodiment, a base three notation height transform technique is utilized rather 

15 than the binary height transformation technique discussed previously. In the base 
three transformation technique, a target ring is designated by a sequence of base 
three numbers. Thus, one a level n, the n low-order base three numbers of the 
height designation are reversed in order, the low-order-bit-reversed height 
designation is incremented by one, and the n low-order base three numbers are 

20 reversed back again. An exemplary interconnect structure has four levels (J =3 
plus one), nine heights (3^ = 3^) per level and five nodes (K=5) per height. In 
accordance with the base three height transformation technique, node N2(2oi)3 
level 2 has a first data input terminal connected to a data output terminal of node 
^3(20 1)2 on level 3, a second data input terminal connected to a data output terminal 

25 of node N2(220)2 on level two. Node N2(2oi)3 ^^so has a first data output terminal 
connected to a data input termmal of node Ni(2oi)4 on level one and a second data 
output terminal coimected to a data input terminal of node J^icinyi' Node N2(2oi)3 
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also has a control input bit connected to a control output bit of node Ni(2oo)3 
a control output bit connected to a control input bit of node N3(2ii)3- In this 
embodiment, the header includes a synch bit followed by the address of a target 
ring in base three. For example, the base three numbers are symbolized in binary 
5 form as 00, 01 and 10 or using three bits in the form 001, 010 and 100. 

Further additional height transformation techniques are possible using 
various numeric bases, such as base 5 or base 7 arithmetic, and employing the 
number reversal, mcrement and reversal back method discussed previously. 

Multiple Level Step Embodiment 

10 In another embodiment, an interconnect structure of the nodes have ten 

terminals, including five input terminals and five output terminals. The input 
termmals include three data input terminals and two control input terminals. The 
output terminals include three data output terminals and two control output 
termmals. In this interconnect structure, nodes are generally connected among five 

15 adjacent cylindrical levels. Specifically, nodes N(T,6,z) at the T cylindrical level 
have termmals connected to nodes on the T, T+1, T+2, T-1 and T-2 levels. 
These coimections are such that the nodes N(T,^,z) have data input termmals 
connected to nodes on the same level T, the next outer level T+ 1 and the previous 
outer level T+2. In particular, nodes N(T,^,z) have data input terminals 

20 connected to data output terminals of nodes l^(J,e-l,hr^z)), N(T+l,0-l,z) and 
N(T+2,0-l,z). Nodes N(T,e,z) also have control output terminals, which 
correspond to data input terminals, connected to nodes on the next outer level T+ 1 
and the previous outer level T+2. Nodes N(T,0,z) have control output terminals 
connected to control mput terminals of nodes N(T+l,0-l,z) and N(T+2,5-l,z). 

25 Nodes N(T,0,z) also have data output terminals connected to nodes on the same 
level T, the next inner level T-1 and the subsequent inner level T-2. In particular, 
nodes N(T,5,z) have data output terminals connected to data output terminals of 
nodes N(T.e + l,HT<z)), N(T-l,^+l,z) and N(T-2,^+l,z). Nodes N(T,a,z) also 
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have control input terminals, which correspond to data output terminals, connected 
to nodes on the next inner level T-1 and the subsequent inner level T-2. Nodes 
N(T,0,z) have control input terminals connected to control output terminals of 
nodes N(T-l,0+l,z) and N(T-2,(9+l,z). 

5 This ten-terminal structure applies only to nodes at the intermediate levels 

2 to J-2 since nodes at the outer levels J and J-1 and at the inner levels 1 and 0 
have the same connections as the standard six-terminal nodes. 

This ten-terminal strucmre allows messages to skip past levels when 
possible and thereby pass through fewer nodes at the cost of increasing logic at the 

10 nodes. Only one message is allowed to enter a node at one time. The priority of 
message access to a node is that a message on the same level has top priority, a 
message from a node one level removed has second priority and a message from 
a node two levels away has last priority. Messages descend two levels whenever 
possible. The timing rules for an interconnect structure using the six-terminal 

15 nodes. Advantages of the ten-terminal node interconnect structure are that 
messages pass more quickly through the levels. 

Other interconnect structure embodiments include nodes having more than 
ten terminals so that data and control terminals are connected to additional nodes 
on additional levels. For example, various nodes N(T,^,z) also have associated 

20 control input termmals and data output terminals, which are connected to nodes on 
inner levels T-3, T-4 and so on. In other examples, various nodes N(T,0,z) also 
have associated control output terminals and data input terminals, which are 
connected to nodes on outer levels T+3, T+4 and so on. In various interconnect 
structure embodiments, nodes may be connected among all levels or selected 

25 levels. 

Multiple Interconnections to the Same Level Embodiment 
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Additional interconnect stracture embodiments utilize additional 
interconnections among nodes on the same level. Specifically, nodes N(T,0,z) on 
the level T have interconnections in addition to the connections of (1) an output 
data terminal connected to an input data terminal of nodes N(T,^-t- l,hj(z)) and (2) 
5 an input data terminal connected to an output data terminal of nodes N(T,6- 
l,Hj(z)). Thus nodes N(T,0,z) on the level T have interconnections including a 
connection of (1) an output data terminal connected to an input data terminal of 
nodes N(T,0 + l,g-j<z)) and (2) an input data terminal connected to an output data 
terminal of nodes N(T,^-l,br(z)). Like cylinder height h-^Kz), height gx(z) is on 
10 the half of the interconnect structure of level T that is opposite to the position of 
height z (meaning bit T of the binary code describing height h-i<z) and gx(z) is 
complementary to bit T of height z). 

Multiple Interconnections to a Next Level Embodiment 

A multiple interconnections to a next level embodiment is similar to the 
15 multiple interconnections to the same level embodiment except that node N(T,^,z) 
has one output data terminal connected to one node N(T,0 4- l,h-p(z)) on level T and 
two output data termmals connected to two nodes N(T-1.0-I-I,z) and N(T- 
l,^-f-l,gj.i(z)) on level T-I. Thus one data ouQjut interconnection traverses the 
same level, a second interconnection progresses one level and a third 
20 interconnection both progresses one level and traverses. Like height hrCz), height 
gT(z) is on the half of the mterconnect structure of level T that is opposite to the 
position of height z. Conflicts between node access are resolved by applying a 
first priority to messages moving on the same level, a second priority, to messages 
progressing one level and a third priority to messages both progressing one level 
25 and traversing. 

The description of certain embodiments of this invention is intended to be 
illustrative and not limitmg. Numerous other embodunents will be apparent to 
those skilled in the art, all of which are included within the broad scope of this 
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invention. For example, many different types of devices may be connected using 
the interconnect structure including, but not limited to, workstations, computers, 
terminals, ATM machines, elements of a national flight control system and the 
like. Also, other interconnection transformations other than and H-j- may be 
5 implemented to describe the interconnections between nodes. 



The description and claims occasionally make reference to an interconnect 
strucmre which is arranged in multiple dimensions. This reference to dimensions 
is useful for understanding the interconnect structure topology. However, these 
dimensions are not limited to spatial dimensions but generally refer to groups of 
10 nodes which are interconnected in a particular manner. 



